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PREFACE 


This  Note  assesses  the  impact  of  nonresponse  in  baseline  data  collection  on  the 
ability  to  analyze  treatment  effects  in  Project  ALERT  (Adolescent  Experiences  in 
Resistance  Training),  a  multisite,  multiyear  test  of  a  smoking  and  drug  prevention 
program  for  adolescents.  The  project  was  supported  by  the  Conrad  N.  Hilton 
Foundation. 

The  Note  should  interest  researchers  who  wish  to  evaluate  the  quality  of  the 
Project  ALERT  database,  as  well  as  those  who  plan  to  undertake  similar  research  on 
school-based  populations.  The  design  and  implementation  of  Project  ALERT  is 
described  in  Ellickson  et  al.  (1988).  Results  from  the  program’s  first  IS  months  appear  in 
Ellickson  and  Bell  (1990). 


This  Note  evaluates  the  impact  of  baseline  nonresponse  in  Project  ALERT 
(Adolescent  Experiences  in  Resistance  Training),  a  longitudinal  experiment  that  tests  the 
effectiveness  of  a  school-based  drug  prevention  program  for  seventh-  and  eighth-grade 
students  in  20  treatment  and  10  control  schools  in  California  and  Oregon.  Of  the 
intended  sample,16  percent  failed  to  participate  in  some  or  all  of  the  baseline  data 
collection.  The  main  reasons  were:  parents’  refusal  to  allow  their  children  to  participate 
in  any  data  collection,  students’  refusal  to  participate  in  some  or  all  of  the  data  collection, 
and  absenteeism  from  school  on  survey  dates.  Students  who  missed  the  baseline  survey 
will  be  dropped  from  the  main  analysis  of  treatment  effects  in  Project  ALERT. 

This  analysis  addresses  four  questions: 

1 .  How  many  students  were  lost  because  of  baseline  nonresponse? 

2.  How  did  each  group  of  nonrespondents — parent  refusals,  student  refusals, 
and  absentees — differ  from  respondents? 

3.  Overall,  how  did  nonresponse  change  the  sample? 

4.  Did  nonresponse  introduce  differences  among  the  treatment  groups? 

The  last  two  questions  bear  directly  on  the  validity  of  the  results  on  treatment 
effectiveness. 

Earlier  prevention  studies  have  found  that  each  nonresponse  group  includes 
disproportionately  more  students  who  have  used  substances  or  who  were  at  high  risk  to 
do  so.  To  the  extent  that  nonrespondents  differ  from  a  random  sample  of  the  original 
population,  the  losses  alter  the  sample  available  for  analysis  and  limit  the  experiment’s 
external  validity.  Furthermore,  if  nonresponse  differs  by  treatment,  it  can  threaten 
internal  validity  (whether  an  experiment  provides  unbiased  information  about  treatment 
effects  for  the  sample  that  remains). 
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METHODS 

To  evaluate  the  program’s  effectiveness,  we  asked  students  to  fill  out 
nuestionnaires  about  their  drug  use  and  related  topics  and  to  provide  samples  of  saliva. 

To  reduce  the  impact  of  baseline  nonresponse,  we  chose  passive  informed  consent, 
which  requires  a  parent  to  return  a  (non)consent  form  only  if  permission  is  denied.  We 
also  recaptured  absentees  in  18  schools  by  using  makeup  data  collection  sessions. 

We  used  several  data  sources  in  this  analysis  to  compare  respondents  with  each 
type  of  nonrespondent  on  various  institutional,  demographic,  and  personal 
characteristics.  Because  survey  data  were  unavailable  for  students  whose  parents 
refused  consent,  we  turned  to  two  other  data  sources,  student  master  lists  and  school 
records.  The  samples  for  these  special  data  collections  included  all  the  nonrespondents 
and  a  stratified  random  sample  of  SO  respondents  from  each  school. 

RESULTS 

Some  8.5  percent  of  parents  denied  consent,  accounting  for  the  majority  of 
baseline  nonresponse.  Slightly  less  than  1  percent  of  students  refused  to  fill  out  a  survey 
or  provide  a  saliva  sample.  Approximately  2  percent  responded  to  the  survey  but  refused 
to  provide  saliva,  and  3  percent  were  absent.  Miscellaneous  other  nonresponse  raised  die 
total  to  16  percent  of  the  baseline  population. 

Each  type  of  nonresponse  related  significantly  to  school,  but  only  student  refusals 
related  to  district.  Student  refusals  were  also  influenced  by  what  might  be  termed  a 
“bandwagon"  effect:  Once  one  or  more  students  had  refused  part  of  the  data  collection, 
other  students  in  the  same  class  were  likely  to  refuse  too. 

Baseline  nonrespondents  because  of  absence  and  student  refusals  included  more 
students  at  high  risk  for  future  substance  use.  Both  groups  had  much  lower  grades  and 
many  more  absences  than  respondents  did.  Postbaseline  survey  data  also  showed 
significantly  higher  use  of  marijuana  and  cigarettes  by  both  groups. 

Compared  to  respondents,  students  whose  parents  refused  for  diem  tended  to  be  at 
similar  or  less  risk  for  future  substance  use.  This  finding  differs  dramatically  from  what 
others  have  found  for  refusals  of  active  parent  consent  Students  in  the  parent  refusal 
group  in  Project  ALERT  had  higher  average  grades  than  respondents  did,  although  the 
differences  were  not  significant.  Numbers  of  absences  were  almost  identical  to  those  for 
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respondents.  In  each  case,  the  results  differed  significantly  from  those  for  each  of  the 
other  nonresponse  groups. 

Despite  substantial  differences  in  respondents  from  absentee  and  student  refusal 
groups,  baseline  nonresponse  altered  sample  characteristics  only  slightly.  For  example, 
we  estimate  that  nonresponse  increased  the  mean  math  grade  point  average  by  only  0.02 
grade  points — from  2.39  in  the  baseline  population  to  2.41  among  respondents.  The 
reason  is  that  total  nonresponse  was  low,  and  the  largest  nonresponse  group — parent 
refusals — tended  to  resemble  respondents.  That  is,  parent  refusals  tended  to  dampen  the 
impact  of  other  nonresponse  rather  than  to  aggravate  it. 

Parents  and  students  knew  the  treatment  assignments  before  baseline,  but  that 
knowledge  does  not  seem  to  have  affected  who  responded.  Although  each  type  of 
nonresponse  occurred  most  in  the  control  condition,  none  of  the  individual  differences 
achieved  statistical  significance.  More  important,  we  found  no  evidence  that  treatment 
affected  who  failed  to  respond.  Thus,  no  evidence  exists  that  baseline  nonresponse 
altered  the  balance  among  the  treatment  groups. 

The  steps  taken  to  reduce  baseline  nonresponse  in  Project  ALERT — passive 
informed  consent,  careful  explanation  of  data  privacy,  and  makeup  sessions  for 
absentees — proved  successful.  They  held  the  combined  level  of  baseline  nonresponse  to 
16  percent.  More  important,  these  steps  sharply  limited  differences  between  the  baseline 
population  and  baseline  sample  and  avoided  substantial  differences  among  the  treatment 
groups. 
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I.  BASEUNE  NONRESPONSE  IN  PROJECT  ALERT 


INTRODUCTION 

In  an  ideal  world,  all  subjects  selected  for  a  research  project  would  actually 
participate.  Inevitably,  however,  some  potential  subjects  fail  to  respond.  This  Note 
evaluates  the  impact  of  baseline  nonresponse  in  Project  ALERT  (Adolescent  Experiences 
in  Resistance  Training) — a  multisite,  longitudinal  experiment  designed  to  evaluate  a 
school-based  drug  prevention  program.1  The  program,  targeted  at  seventh-  and  eighth- 
grade  students,  seeks  to  prevent  or  delay  drug  use  onset  and  the  transition  to  regular  use 
or  abuse. 

Students  who  missed  the  baseline  survey  will  be  dropped  from  the  main  analysis 
of  treatment  effects  in  Project  ALERT.  That  analysis  requires  baseline  substance-use 
data  to  assign  students  to  analysis  subgroups  based  on  exposure  to  the  three  target 
substances:  cigarettes,  alcohol,  and  marijuana  (Ellickson  and  Bell,  1990).  The  main 
reasons  for  baseline  nonresponse  were:  parents’  refusal  to  allow  their  children  to 
participate  in  any  data  collection,  students’  refusal  to  participate  in  some  or  all  of  the  data 
collection,  and  absenteeism  from  school  on  survey  dates. 

Our  analysis  addresses  four  questions: 

1 .  How  many  students  faded  to  respond  at  baseline? 

2.  How  did  each  group  of  nonrespondents — parent  refusals,  student  refusals, 
and  absentees — differ  from  respondents? 

3.  Overall,  how  did  nonresponse  change  the  sample? 

4.  Did  nonresponse  introduce  differences  among  the  treatment  groups? 

The  last  two  questions  bear  directly  on  the  validity  of  the  results  on  treatment 
effectiveness.  If  nonresponse  systematically  eliminated  from  die  experiment  certain 
types  of  students  (for  example,  those  most  likely  to  use  drop),  this  would  alter  the 
population  to  which  the  results  applied.  Similarly,  if  nonresponse  created  substantial 

1This  Note  does  not  address  problems  associated  with  attrition  after  baseline  or  other 
sources  of  missing  data  (see  Ellickson  et  al.,  1988). 
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imbalance  among  the  treatment  groups,  it  could  undeimine  the  results.  Using  various 
data  sources  to  compare  baseline  nonrespondents  with  baseline  respondents,  we  examine 
these  two  potential  threats  to  the  experiment  The  analysis  also  provides  evidence  about 
the  impact  of  steps  we  took  to  reduce  nonresponse. 

OVERVIEW  OF  PROJECT  ALERT 

Project  ALERT  took  place  in  30  diverse  schools  in  eight  California  and  Oregon 
districts.  The  schools  are  located  in  rural,  suburban,  and  urban  communities  with  a 
variety  of  socioeconomic  and  racial  compositions.  Nine  schools  had  minority 
populations  of  SO  percent  or  more.  This  is  important  because  most  previous  evaluations 
have  failed  to  include  schools  with  substantial  minority  populations  (Botvin  and  Wills, 
1985). 

During  the  1984-1985  academic  year,  eight  weekly  lessons  were  taught  to  all 
seventh-grade  students  in  20  of  the  30  schools.  In  10  schools,  assigned  at  random,  an 
outside  health  educator  alone  presented  the  curriculum,  denoted  the  Health  Educator 
Curriculum  (HEC);  in  the  other  10  treatment  schools,  two  local  high  school  students 
assisted  the  health  educator  in  four  of  the  eight  sessions,  denoted  the  Teen  Leader 
Curriculum  (TLC).  The  10  other  schools,  which  received  no  special  curriculum,  served 
as  the  experiment’s  control  group. 

To  evaluate  the  program’s  effectiveness,  we  asked  students  to  fill  out 
questionnaires  about  their  drug  use  and  related  topics  and  to  provide  specimens  of  saliva. 
Students  were  told  that  tobacco  and  marijuana  use  can  be  detected  in  saliva  and  that  the 
specimens  would  be  tested.  At  the  beginning  of  each  data  collection  session,  students 
received  the  option  to  refuse  to  participate  in  either  or  both  types  of  data  collectioa 

Baseline  data  collection  was  conducted  the  week  before  curriculum  sessions 
began  in  a  district  Program  implementation  occurred  in  two  phases,  each  contained 
within  either  the  fall  or  spring  semester.  At  12  schools  in  four  districts,  baseline  data 
collection  occurred  during  the  first  two  weeks  of  October  1984.  At  the  1 8  schools  in  the 
other  four  districts,  baseline  data  collection  began  near  the  end  of  February  1985.  By 
April  1988,  a  total  of  six  data  collection  waves  were  completed.  Ellickson  et  al.  (1988) 
more  fully  describe  the  curriculum,  experimental  design,  and  data  collection. 

For  analyses  of  the  effectiveness  of  the  Project  ALERT  curriculum,  we  have 
identified  what  we  term  the  experiment’s  baseline  population  of  7566  students.  The 
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baseline  population  includes  all  seventh-grade  students  who  were  enrolled  in  mainstream 
classes  in  a  project  school  at  baseline  and  who  remained  in  the  same  school  through 
week  3  of  the  seventh-grade  curriculum  (for  the  core  of  the  curriculum). 

POTENTIAL  IMPACT  OF  BASEUNE  NONRESPONSE  ON  EXPERIMENTAL 
RESULTS 

The  most  obvious  effect  of  baseline  nonresponse  is  to  reduce  the  sample  size 
available  for  analysis.  Some  prevention  studies  have  lost  about  half  their  potential 
students.  Such  losses  substantially  increase  the  standard  errors  of  treatment  effect 
estimates  and  limit  the  opportunity  to  observe  statistical  significance.  However,  losses  of 
that  magnitude  cause  a  much  more  important  problem:  They  threaten  an  experiment’s 
validity. 

External  Validity 

The  value  of  any  study  depends  on  whether  the  results  apply  to  a  population  of 
interest.  External  validity  refers  to  the  extent  that  the  results  of  an  experiment  apply 
beyond  the  particular  sample. 

Nonresponse  can  reduce  external  validity.  When  nonrespondents  differ  from  a 
random  sample  of  the  baseline  population,  the  losses  change  the  sample  available  for 
analysis.  If  the  baseline  population  (the  sample  before  nonresponse)  was  carefully 
chosen  to  match  the  target  population  for  an  intervention,  then  the  change  would  reduce 
the  external  validity.  The  size  of  that  change  depends  on  two  factors:  the  overall 
nonresponse  rate  and  the  amount  of  difference  between  respondents  and  all 
nomespondents.  If  either  factor  can  be  kept  small,  then  the  impact  of  nonresponse  will 
also  remain  small. 

On  the  other  hand,  small  or  moderate  changes  to  the  baseline  sample  do  not 
necessarily  imply  a  loss  of  external  validity.  External  validity  rests  on  the  characteristics 
of  the  final  sample,  which  depend  mainly  on  the  initial  selection  of  subjects.  If  a  study 
includes  only  upper-class  suburban  schools,  one  should  have  little  confidence  that  die 
results  will  also  apply  to  students  in  other  environments.  Project  ALERT  was  not 
designed  as  a  probability  sample  of  all  seventh-grade  students  in  the  United  States,  or  in 
California  and  Oregon;  no  sample  of  eight  districts  could  adequately  serve  that  purpose. 
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Instead,  our  goal  was  to  include  a  variety  of  schools  whose  students  represent  the 
diversity  of  the  U.S.  population. 

Losses  from  substance-use  prevention  experiments  are  often  thought  to  include 
disproportionately  more  students  at  high  risk  for  substance  use  (Biglan  and  Ary,  1985). 

In  that  case,  experiments  would  provide  less  information  about  the  part  of  the  population 
most  needing  help. 

Past  studies  have  indicated  that  students  whose  parents  refuse  to  provide  active 
informed  consent  differ  substantially  from  those  whose  parents  do.  Active  consent 
requires  that  a  parent  return  a  signed  consent  form  permitting  the  student  to  participate  in 
the  experiment.  If  the  form  is  not  signed  and  returned,  the  student  must  be  excluded 
from  data  collection.  Active  consent  typically  causes  a  loss  of  40  to  50  percent  of  all 
potential  students  (Josephson  and  Rosen,  1978;  Severson  and  Ary,  1983),  including 
relatively  more  blacks,  Asian  Americans,  low  achievers,  and  children  with  less 
well-educated  parents  (Kearney  et  al.,  1983;  Lueptow  et  al.,  1977;  Severson  and  Ary, 
1983;  Thompson,  1984). 

To  avoid  the  high  nonresponse  rate  typical  of  active  consent,  many  researchers 
have  turned  to  passive  informed  consent,  which  requires  a  parent  to  return  a 
(non)consent  form  only  if  permission  is  denied  (Biglan  et  al.,  1987;  Murray  et  al.,  1987). 
If  the  form  is  not  returned,  the  student  can  participate  in  data  collection.2  However,  little 
is  known  about  how  passive  consent  affects  a  sample’s  characteristics.  We  used  passive 
consent  in  Project  ALERT.3  Hence,  this  study  provides  new  information  about  whether 
the  two  consent  procedures  yield  similar  results. 

Some  students  who  refuse  to  fill  out  the  questionnaire  or  to  provide  saliva  might 
do  so  to  conceal  whether  they  use  drugs.  If  that  happened,  we  would  expea  refusals  to 

^The  danger  of  passive  consent  is  that  parents  who  do  nothing  (consent  passively)  may 
not  understand  the  consequences  of  their  inaction.  If  many  of  those  parents  do  not  want 
their  children  to  participate  in  data  collection,  passive  consent  would  not  provide  true 
informed  consent.  To  leant  whether  passive  consent  would  fail  in  that  respect,  Eilickson 
and  Hawes  (1989)  conducted  a  pilot  study  as  part  of  Projea  ALERT’S  development  of 
data  collection  procedures.  That  study  indicated  that  passive  consent,  combined  with 
diligent  efforts  to  reach  and  inform  parents,  can  provide  adequate  informed  consent  for 
data  collection  of  this  type.  Severson  and  Biglan  (1989)  readied  the  same  conclusioa 
h'o  maximize  the  chances  that  parents  would  receive  and  read  the  informed  consent 
material,  we  provided  three  separate  deliveries;  an  initial  package  via  first-class  mail,  a 
reminder  postcard  one  week  later,  and  a  repeat  of  the  initial  package  taken  home  by 
students  three  weeks  after  the  initial  mailing  (Eilickson,  1989). 


lower  the  reported  use  rates.  However,  students  might  refuse  for  other  reasons  not 
relating  clearly  to  drug  use.  For  example,  some  students  in  Project  ALERT  refused  to 
provide  saliva,  complaining  that  it  was  “gross."  If  that  reason  accounted  for  all  saliva 
refusals,  those  students  might  differ  from  respondents  only  in  their  degree  of 
assertiveness. 

Although  absences  tend  to  occur  at  random  times,  the  absentees  on  any  given  day 
certainly  do  not  represent  the  population  of  all  students.  Several  studies  have  found  that 
frequent  absences  predict  subsequent  drug  use;  hence,  absentees  would  presumably 
include  a  greater  percentage  of  substance  users  than  would  respondents. 

Internal  Validity 

Internal  validity  means  that  an  experiment  provides  unbiased  information  about 
treatment  effects  for  the  sample  that  remains.  This  requires  that  the  sets  of  respondents 
forming  the  treatment  groups  resemble  each  other.  Differences  could  arise  in  either  of 
two  ways;  if  nonresponse  rates  differ  among  treatments,  or  if  the  type  of  students  who 
fail  to  respond  differs  among  treatments.4 

Because  parents  and  students  knew  the  treatment  assignments  at  the  time  of 
informed  consent  and  baseline  data  collection,  treatment  could  have  affected  the 
frequency  of  nonresponse.3  Parents  of  students  in  the  treatment  groups  might  refuse  less 
often  because  their  children  were  receiving  something  valuable  in  return,  or  more  often 
because  some  did  not  want  their  children  to  receive  the  curriculum.  To  avoid  the  latter 
occurrence,  the  informed  consent  material  emphasized  that  the  schools  mandated 
participation  in  the  curriculum  and  that  refusing  consent  applied  only  to  data  collection. 
How  treatment  might  affect  which  type  of  student  fails  to  respond  is  less  clear.  Although 
there  is  less  reason  to  expect  these  problems  than  ones  affecting  external  validity,  our 
analysis  investigates  them  as  well. 


4An  even  larger  threat  to  internal  validity  would  occur  if  an  entire  school  were  lost 
from  the  experiment  after  completion  of  the  treatment  assignment  This  did  not  happen 
in  Project  ALERT. 

sNecessary  preparations  for  providing  the  curriculum  in  treatment  schools  precluded 
concealing  the  assignments  from  parents  or  students. 
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DISTRIBUTION  OF  BASEUNE  NONRESPONSE 

We  took  three  steps  to  reduce  baseline  nonresponse  in  Project  ALERT.  First,  we 
used  passive  parent  consent.  Second,  in  the  18  schools  where  the  program  was 
implemented  in  the  spring  semester,  we  conducted  makeup  sessions  for  students  who 
missed  school  on  the  day  of  baseline  data  collection  scheduled  for  their  class.  Third,  to 
reduce  the  motivation  for  students  to  refuse  (or  to  lie),  we  told  the  students  that  no  one  at 
their  school  or  home  would  see  any  of  their  answers  and  explained  our  procedures  for 
ensuring  this. 

The  overall  nonresponse  rate  was  16  percent  of  the  baseline  population — divided 
into  six  types  (see  Table  1).  Some  8.5  percent  of  parents  denied  consent,  accounting  for 
the  majority  of  baseline  nonresponse.  Slightly  more  than  3  percent  were  absent  on  the 
day  of  data  collection  and  the  makeup  day  (in  spring  schools).  Less  than  1  percent  of 
students  refused  to  fill  out  a  survey  or  provide  a  saliva  sample;  approximately  2  percent 
responded  to  the  survey  but  refused  to  provide  saliva.  Because  we  have  baseline  surveys 
for  students  who  refused  saliva  only,  those  students  will  be  included  in  the  experiment's 
evaluation  of  treatment  effects.  We  count  them  as  nonrespondents  in  this  analysis 
because  we  wanted  to  compare  them  with  complete  respondents  and  we  could  not  always 
analyze  survey  refusals  separately  (see  Sec.  II). 

The  fifth  group— parent  consent  not  solicited — consists  mainly  of  students  who 
enrolled  in  a  Project  ALERT  school  within  a  week  or  two  before  baseline,  too  late  to 


Table  1 


DISTRIBUTION  ACROSS  NONRESPONSE  CATEGORIES 
IN  THE  BASELINE  POPULATION 


Nonresponse  Category 

Number 

Percentage  of 

Baseline  Pop.  Nonresponse 

Parent  refusals 

642 

8.5 

52.9 

Student  refusal  of  survey 

58 

.8 

4.8 

Student  refusal,  saliva  only 

171 

2.3 

14.1 

Student  absent 

246 

3.3 

20.3 

Parent  consent  not  solicited 

79 

1.0 

6.5 

Miscellaneous  nonresponse 

17 

2 

1.4 

Total  nonresponse 

1213 

16.1 

100.0 

Total  response 

6353 

83.9 

(a) 

Total  baseline  population 

7566 

100.0 

(a) 

•Not  applicable. 
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allow  their  parents  adequate  time  to  give  informed  consent  The  last  group — 
miscellaneous  nonresponse — includes  students  who  left  die  classroom  for  reasons  that 
did  not  relate  to  refusal.  Because  these  groups  are  small  and  would  not  be  expected  to 
differ  greatly  from  respondents,  we  have  not  included  their  members  in  the  nonresponse 
analysis. 

The  steps  taken  to  reduce  baseline  nonresponse  worked  very  well.  The  rate  of 
parent  refusals  was  a  fraction  of  that  in  some  studies  that  used  active  consent.  The  low 
absentee  rate  results  from  the  makeup  sessions,  which  reduced  die  absentee  rate  at  spring 
baseline  from  8.3  percent  before  makeups  to  2.5  percent  At  fall  baseline,  the  absentee 
rate  was  5  percent  without  makeups.  Student  refusals  of  the  experiment’s  main  data 
source — the  survey — were  negligible. 


J 
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II.  DATA  SOURCES  AND  DATA  COLLECTION  METHODS 


Several  data  sources  allowed  us  to  compare  respondents  with  each  type  of 
nonrespondent  on  institutional,  demographic,  and  personal  characteristics.  We  used 
sample  accounting  data  to  relate  nonresponse  to  district,  school,  class,  and  treatment 
Postbaseline  survey  data  provided  substance  use  information  for  many  of  the  student 
refusals,  absentees,  and  respondents. 

Because  survey  data  were  unavailable  for  students  whose  parents  refused  consent, 
we  turned  to  two  other  data  sources:  (1)  our  student  master  lists,  which  contained  gender 
and  race/ethnicity  data  for  both  respondents  and  nonrespondents;  and  (2)  school  records, 
from  which  we  could  abstract  academic  grades  and  absences  for  both  groups.  The 
school  records  data  provided  the  best  available  indicators  of  substance  use. 

SAMPLE  FOR  DEMOGRAPHIC  AND  ACADEMIC  COMPARISONS 

To  compare  nonrespondents  and  respondents  with  respect  to  grades,  absenteeism, 
race/ethnicity,  and  gender,  we  constructed  special  data  sets — one  for  grades  and 
absences,  the  other  for  race  and  gender.  We  initially  selected  2771  students  for  inclusion 
in  both  analyses.  This  group  included  a  stratified  random  sample  of  SO  respondents  from 
each  school  (if  available)  and  all  the  nonrespondents  in  each  of  three  categories — parent 
refusals,  student  refusals,  and  absentees.  Fifty  respondents  per  school  far  exceeded  the 
number  in  any  nonresponse  category  at  most  schools;  sampling  more  respondents  would 
have  increased  data  collection/entry  costs  without  substantially  improving  precision.  To 
improve  correspondence  between  the  sample  and  the  population  of  respondents  from 
each  school,  we  selected  stratified  random  samples  from  each  school.1  For  fall  schools, 
the  stratifying  variable  was  a  questionnaire  item  eliciting  information  about  the  student’s 
usual  grades.  Because  responses  to  this  item  were  not  available  in  time  for  selection  of 
the  spring  sample,  we  stratified  by  data  collection  session. 


'This  sampling  procedure  is  described  by  Chromy  (1979).  Subject  to  randomization, 
the  procedure  provides  the  closest  possible  match  of  the  distribution  of  the  stratifying 
variable  in  die  sample  to  its  distribution  in  the  population. 
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This  initial  sample  contained  1484  respondents  and  1287  nonrespondents. 

Because  we  needed  to  select  this  sample  before  identifying  the  baseline  population,  it 
included  many  students  we  had  attempted  to  survey  but  who  were  ineligible  for  our 
baseline  population.  These  include  eighth  graders,  special  education  students,  students 
who  transferred  within  five  weeks  of  baseline,  and  some  students  who  appeared  on 
enrollment  lists  but  were  not  enrolled  at  baseline.  Deleting  these  students  required 
special  procedures  for  each  data  set 

GENDER  AND  RACE/ETHNICITY  DATA 

We  abstracted  the  gender  and  race/ethnicity  data  from  printed  school  master  lists, 
aggregating  different  district  codes  to  a  consistent  set  of  ethnic  categories:  white,  black, 
Hispanic,  Pacific  Islander/Asian,  and  American  Indian.  We  deleted  five  sensitive 
cases — instances  where  only  one  or  two  students  fell  into  a  school’s  particular 
nonresponse  category.  If  these  cases  had  remained  in  the  data  set,  individual  students 
might  have  been  readily  identifiable  even  without  access  to  the  link  between  name  and 
identification  number.  We  also  identified  and  deleted  167  students  who  were  not  part  of 
the  baseline  population — eighth  graders,  special  education  students,  and  transfer 
students.2  Finally,  we  could  not  find  18  cases  in  the  master  lists.  The  gender  and 
race/ethnicity  analysis  data  set  contained  2581  cases  (see  Table  2). 

GRADES  AND  ABSENCE  DATA 

From  school  records,  we  collected  data  on  fall  semester  absences  and  final  fall 
semester  grades  in  seventh-grade  English  and  math.  We  used  fall  semester  records  to 
allow  completion  of  data  collection  before  the  1985  summer  break.  Where  available, 
unexcused  and  excused  absences  were  counted  separately.  However,  because  the 
participating  schools  use  different  standards  for  defining  unexcused  absences,  we  report 
results  for  total  absences  only. 

Most  schools  did  not  want  to  provide  grades  and  absence  information  by  name, 
particularly  for  students  whose  parents  had  refused  consent  for  participation  in  the  study. 
Consequently,  we  developed  a  special  data  collection  form  to  keep  student  data 

2Because  some  ineligible  students  were  not  identified  until  after  creation  of  this  data 
set,  it  includes  some  students  who  are  not  in  the  baseline  population. 
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Table  2 

REASONS  FOR  DELETION  OF  STUDENTS  FROM  SAMPLES 
FOR  THE  NONRESPONSE  ANALYSES 


Type 


Category 

Gender  and 
Race/Ethnicity 

Grades  and 
Absences 

Deletions 

Sensitive  cases 

5 

5 

Identified  as  ineligible* 

167 

87 

School  with  many  eighth  graders 

0 

118 

Not  in  records 

18 

22 

Total  deletions 

190 

232 

Initial  sample 

2771 

2771 

Total  available  for  analysis 

2581 

2539 

"Not  in  baseline  population. 


confidential.  Each  page  contained  three  columns,  with  student  names  in  the  far  left 
column.  Next  to  each  name  was  a  removable  label  where  the  grades  and  absence  data 
were  to  be  entered.  The  labels  identified  whether  the  student  responded  at  baseline  and, 
if  not,  the  appropriate  nonresponse  category.  After  the  data  had  been  copied  from  school 
records  onto  the  labels,  the  labels  were  moved  in  random  order  to  the  far  right  column  of 
the  page.  This  procedure  severed  the  linkage  between  student  name  and  grades/absences 
data  before  the  forms  reached  RAND.  Hence,  we  could  not  link  data  between  the 
demographic  and  academic  data  sets. 

As  before,  we  deleted  fire  five  sensitive  cases  from  the  grades  and  absences  data 
set.  We  also  deleted  22  students  with  data  on  neither  fall  semester  grades  nor  absences. 

Identifying  ineligible  students  was  considerably  more  difficult  because  we  could 
not  use  names  to  link  grades/absences  data  to  the  project’s  sample  accounting  data.  We 
worried  particularly  about  students  whose  names  appeared  on  enrollment  lists  even 
though  they  were  not  really  enrolled  at  baseline.  When  the  grades/absences  data  were 
collected,  these  students  were  often  classified  as  absent  (instead  of  ineligible).  If  school 
records  showed  these  students  to  have  many  absences  and  failing  grades,  that  would 
distort  our  analysis  of  absentees.  Written  comments  on  labels  enabled  us  to  delete  87 
apparently  ineligible  students  from  13  schools,  and  we  determined  that  there  were  none 
in  6  more  schools.  Other  information  indicated  that  there  were  few  or  no  ineligible 
students  in  10  of  the  other  1 1  schools.  Because  the  school  where  eighth  graders 
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partici paled  in  data  collection  included  some  50  ineligible  students  whom  we  could  not 
identify,  we  deleted  all  1 18  students  in  that  school  from  the  grades  and  absences  analysis. 
The  resulting  grades  and  absences  analysis  file  contained  2539  students  (see  Table  2).3 


Comparing  grades  and  absences  results  restricted  to  the  19  schools  where  we  were 
confident  we  had  deleted  all  ineligible  students  with  results  in  29  schools  indicated  little 
bias  of  the  type  we  had  anticipated.  Thus,  we  present  results  for  the  29  schools. 
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III.  RESULTS 


In  this  section,  we  relate  baseline  nonresponse  rates  to  various  factors.  We  begin 
by  exploring  institutional  factors,  such  as  district,  school,  and  treatment.  Later,  we 
consider  personal  characteristics  that  relate  to  nonresponse.  Except  for  the  comparison 
among  treatment  groups,  these  results  apply  to  die  investigation  of  external  validity — 
whether  nonrespondents  differed  from  respondents.  We  defer  to  Sec.  IV  discussion  of 
how  much  nonresponse  altered  the  baseline  sample. 

INSTITUTIONAL  FACTORS 

All  the  data  presented  in  this  subsection  come  from  the  main  data  collection  files 
for  Project  ALERT  instead  of  from  the  special  data  sets  described  in  Sec.  II.  Thus,  these 
results  refer  to  the  complete  baseline  population  of  7566  students,  less  79  cases  where 
parent  consent  was  not  solicited  and  17  miscellaneous  cases  of  nonresponse  (see 
Table  1).  Because  the  main  Project  ALERT  data  sets  distinguish  student  refusals  of  the 
survey  from  student  refusals  of  saliva  only,  we  consider  four  nonresponse  categories: 
parent  refusals,  student  refusals  of  survey,  student  refusals  of  saliva  only,  and  absentees. 

District  and  School 

Nonresponse  relates  significantly  to  school  but,  for  the  most  part,  not  to  district 
Rates  for  each  type  of  nonresponse  differ  significantly  among  the  30  schools  (P  <  .001 
for  each  type  of  nonresponse,  with  or  without  controlling  for  district).  Analysis  of  rates 
of  student  saliva  refusals  for  the  30  schools,  with  school  as  the  unit  of  analysis,  shows 
significant  differences  among  the  districts  (P  =  .02),  with  the  highest  rates  in  districts 
with  the  most  minority  students.  H  owever,  analyses  of  other  nonresponse  rates  show  no 
significant  differences  among  the  eight  districts  (each  P  >  .15). 

Several  factors  probably  contribute  to  the  differences  in  nonresponse  rates  among 
schools.  As  we  shall  see  below,  homogeneity  of  student  and  parent  characteristics  within 
schools  explains  some  of  the  differences.  In  addition,  short-term  absentee  rates  certainly 
relate  to  illnesses  that  pass  through  schools.  Finally,  decisions  within  the  same  school  to 
refuse  may  depend  on  each  other. 
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Student  refusals  were  clearly  influenced  by  what  might  be  termed  a  "bandwagon" 
effect;  once  one  or  more  students  had  refused  part  of  the  data  collection,  it  became  easier 
for  other  students  in  the  same  class  to  refuse  too.  The  data  collectors  felt  that  they 
observed  this  phenomenon,  and  the  data  support  it  Between-classroom  variation  in  the 
number  of  student  refusals  was  2.7  times  what  would  be  expected  if  all  refusals  were 
independent  within  schools  (true  for  both  survey  and  saliva  refusals).  For  example,  S 
classrooms  (out  of  293),  with  6  to  1 1  refusals  each,  accounted  for  40  of  the  171  saliva- 
only  refusals.  If  refusals  occurred  independently,  that  any  classroom  would  have  had  6 
or  more  is  unlikely.  Perhaps  some  clustering  of  refusals  is  explained  by  correlation  of 
personal  characteristics,  but  the  main  reason  is  probably  that  refusing  becomes  more 
attractive  when  one  sees  others  doing  so. 

Treatment 

Treatment  does  not  seem  to  have  affected  who  responded  at  baseline.  The 
treatments  differed  moderately  in  nonresponse  rates,  but  the  differences  appeared  to 
result  because  each  school  was  assigned  to  a  single  treatment  Although  each  type  of 
nonresponse  was  highest  in  the  control  condition,  none  of  the  individual  differences 
achieved  statistical  significance  based  on  school  as  the  unit  of  analysis  (see  Table  3). 
Statistical  significance  arises  only  when  comparing  the  overall  nonresponse  rate  in  the 
control  group  with  that  in  the  other  two  conditions  combined  (P  =  .025).  Given  the  many 
possible  comparisons,  that  finding  should  be  interpreted  with  caution.  More  important. 


Table  3 

PERCENTAGE  IN  EACH  NONRESPONSE  CATEGORY, 
BY  TREATMENT  GROUP 


Nonresponse  Category 

Treatment  Condition 

TLC  HEC  Control 

P-Value 

Parent  refusal 

8.3 

7.4 

10.2 

.20 

Student  refusal  of  survey 

.5 

.8 

1.0 

.62 

Student  refusal,  saliva  only 

1.4 

2.5 

3.0 

.16 

Student  absent 

2.7 

3.3 

4.0 

.25 

Total  nonresponse 

12.9 

14.0 

18.2 

.08 

NOTE:  P-values  test  the  hypothesis  that  a  particular  type  of 
nonresponse  was  unrelated  to  treatment.  Analysis  of  variance  was 
used  to  compare  school  nonresponse  rates  (weighted  by  school  size) 
across  treatments. 
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we  found  no  evidence  that  treatment  affected  who  failed  to  respond.  Analyses  of  race, 
gender,  English  and  math  grades,  and  absences  showed  that  after  controlling  for  school, 
treatment  did  not  relate  significantly  to  which  types  of  students  responded  (P  = .  16,  .89, 
.96,  .50,  and  .08,  respectively).  In  other  words,  nonrespondents  differed  from 
respondents,  but  appeared  to  do  so  in  the  same  way  in  each  treatment  group. 

PERSONAL  CHARACTERISTICS 

The  analyses  in  this  subsection  use  the  special  data  sets  created  for  assessing 
nonresponse.  Thus,  these  analyses  do  not  distinguish  student  refusals  of  the  survey  from 
refusals  of  saliva  only. 

Race/Ethnldty 

Race/ethnicity  relates  strongly  to  nonresponse  category  (see  Table  4).  Parent 
refusal  rates  were  much  higher  for  Asian  students  than  for  the  other  groups — 13  percent, 
compared  with  8  to  9  percent  Asian  students  were  least  likely  to  be  absent 
compensating  somewhat  for  their  high  rate  of  parent  refusals.  Rates  of  student  refusals 
(survey  combined  with  saliva  only)  were  highest  for  the  Hispanic  students.  Overall, 
baseline  nonresponse  was  highest  for  Hispanic  students  and  lowest  for  whites.  All  the 
relationships  except  those  for  absences  achieved  high  statistical  significance  (first  column 
of  P- values). 


Table  4 


PERCENTAGE  IN  EACH  NONRESPONSE  CATEGORY, 
BY  RACE/ETHNICITY 


Nonresponse  Category 

Race/Ethnicity 

P-Value 

White 

Black 

Hispanic 

Asian 

Pearson 

M-H 

Parent  refusal 

7.9 

8.8 

9.1 

13.1 

.002 

.16 

Student  refusal* 

2.3 

3.7 

6.7 

2.8 

<.001 

.002 

Student  absent 

3.6 

3.6 

4.0 

1.6 

.08 

.33 

Total  nonresponse 

13.8 

16.1 

19.8 

17.5 

<.001 

.18 

NOTE:  P-vslues  teat  the  hypothesis  that  a  particular  type  of  nonresponse  was  unrelated 
to  race.  The  first  act  are  baaed  on  simple  Pearson’s  chi-squares  for  two-way  contingency 
tables.  The  second  set  are  adjusted  for  school  by  use  of  the  general  association  Mantel- 
Haensael  test,  which  uses  only  with  in -school  information  about  the  relationship  of  non¬ 
response  to  race.  American  Indians  are  not  included  because  there  were  too  few  to  provide 
reliable  estimates  of  nonresponse  rates. 

•Combi  nee  refusals  of  survey  and  refusals  of  saliva  only. 
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Because  both  race/ethnicity  and  nonresponse  rates  differ  greatly  among  schools, 
we  tested  for  relationships  between  race  and  nonresponse  within  schools.  The  Mantel- 
Haenszel  test,  which  uses  only  within-school  information,  achieved  statistical 
significance  only  for  student  refusals  (Mantel  and  Haenszel,  1959).  Thus,  we  generally 
could  not  isolate  a  unique  effect  of  race/ethnicity. 

Gender 

Girls  appeared  more  often  than  boys  in  each  nonresponse  group  (see  Table  5). 

The  largest  difference  occurred  for  parent  refusals,  where  girls  were  exempted  from  data 
collection  one-third  more  often  than  boys.  That  difference  and  the  greater  rate  overall 
were  significant  at  P  <  .001. 

Absenteeism 

Students  who  were  absent  frequently  during  fall  semester  were  more  likely  to 
refuse  the  survey  and/or  saliva  than  students  with  few  absences  (see  Table  6).  Not 
surprisingly,  those  students  were  much  more  likely  to  be  absent  on  the  day  of  baseline 
data  collection.1  However,  number  of  absences  did  not  relate  to  the  rate  of  parent 
refusal. 


Tables 

PERCENTAGE  IN  EACH  NONRESPONSE  CATEGORY, 
BY  GENDER 


Gender 


Nonresponse  Category 

Female 

Male 

P-Value 

Parent  refusal 

10.0 

7.4 

<.001 

Student  refusal* 

3.5 

2.7 

.04 

Student  absent 

3.7 

3.2 

.16 

Total  nonresponse 

17.2 

13.3 

<.001 

NOTE:  P-Tahwa  teat  tha  hypothaai*  that  a  particular  type  of  non- 
reaponaa  waa  unrelated  to  gander.  They  are  baaed  on  Pearaon’a  chi- 
aquaraa  for  two-way  contingency  tablea. 

•Combine#  refusals  of  survey  and  refusals  of  saliva  only. 


For  the  fall  sample,  there  is  some  artificial  correlation  resulting  from  die  inclusion  of 
absences  around  baseline  data  collection  in  the  count  of  fall  semester  absences. 
However,  die  relationship  was  almost  as  strong  for  students  in  the  sample  of  spring 
schools. 
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Table  6 

PERCENTAGE  IN  EACH  NONRESPONSE  CATEGORY, 
BY  NUMBER  OF  FALL  SEMESTER  ABSENCES 


Number  of  Absences 


Nonresponse  Category 

0-2 

>2-5 

>5-10 

>10 

P-Value 

Parent  refusal 

9.3 

7.9 

8.1 

9.1 

.46 

Student  refusal* 

2.6 

3.0 

2.8 

4.3 

.02 

Absent 

1.3 

3.4 

4.6 

10.2 

<.001 

Total  nonresponse 

13.2 

14.3 

1S.5 

23.6 

<.001 

NOTE:  P -value*  teat  the  hypotheei*  that  a  particular  type  of  non- 
response  waa  unrelated  to  the  number  of  absences.  They  are  baaed  on 
Pearson’s  chi -squares  for  two-way  eont  .agency  tablea. 

‘Combines  refusals  of  survey  and  refusals  of  saliva  only. 


Grades 

Fall  semester  seventh-grade  English  and  math  grades  also  related  to  nonresponse 
status,  each  showing  a  similar  pattern  (see  Fig.  1).  As  one  might  expect,  students  who 
were  absent  during  data  collection  had  very  low  grades  in  both  English  and  math — 
one-half  grade  point  lower  than  respondents.  To  a  smaller  extent,  the  same  holds  true  for 
student  refusals  (0.23  grade  points  lower  than  respondents).  Questionnaire  data  collected 
from  baseline  absentees  and  student  refusals  at  wave  2  (three  months  after  baseline) 
support  those  in  the  figure.  For  each  of  these  two  nonresponse  groups,  the  proportion 
who  reported  that  their  own  grades  were  usually  C  or  lower  was  52  to  53  percent, 
compared  with  35  percent  for  baseline  respondents. 

In  contrast,  students  whose  parents  refused  for  them  had  higher  grades  than  did 
respondents,  although  die  differences  were  not  statistically  significant  (P  =  .08  for 
English  grades).  Grades  for  the  parent  refusal  group  were  significantly  higher  that  those 
in  either  of  the  other  two  nonresponse  groups.  Because  parent  refusals  retain  that  status 
at  later  waves,  we  could  not  use  wave  2  survey  data  to  confirm  the  finding  of  higher 
grades  for  that  group. 

The  relatively  high  grades  for  students  in  the  parent  refusal  group — the  majority 
of  nonrespondents — dampened  the  overall  difference  between  nonrespondents  and 
respondents.  The  differences  were  0.1 1  and  0.14  for  English  and  math  grades, 
respectively  (P  <  .05  and  P  <  .01). 
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NOTE:  Student  refusals  Include  bo©  refusals  of  the  stwey  and  refusals  of  safiva  only. 
a  Differs  from  mean  for  respondents  at  P  <  .01 . 
bDifters  from  mean  (or  respondents  at  P  <.001. 


Fig.  1 — Mean  English  and  math  grades,  by  nonresponse  category 


SUBSTANCE  USE 

Survey  data  from  wave  2  tend  to  show  higher  substance  use  by  baseline  student 
refusals  and  baseline  absentees  than  by  baseline  respondents  (see  Fig.  2).2  Current 
cigarette  and  marijuana  use  (in  the  past  month)  was  significantly  higher  for  those 
categories  of  baseline  nonrespondents  than  for  baseline  respondents.  The  same  pattern 
occurred  for  alcohol  use,  but  the  differences  were  not  statistically  significant.  No 
substance  use  information  is  available  for  parent  refusals. 


2If  students  who  missed  both  the  first  two  waves  were  especially  likely  to  use 
substances.  Fig.  2  would  understate  the  difference  between  baseline  respondents  and 
these  nonrespondent  groups. 


Percentage 


Alcohol  Cigarettes  Marijuana 


NOTE:  Parent  refusals  do  not  appear  because  no  subetance-use  data  were  collected  tor  them. 
“Differs  from  percentage  ter  respondents  at  P  <  .06. 

^Differs  from  percentage  for  respondents  at  P  <  .001. 


Fig.  2 — Percentage  of  current  substance  users  at  wave  2,  by  nonresponse  category 
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IV.  CONCLUSIONS 


1.  Nonrespondents  and  respondents  differed  substantially.  Our 

analyses  demonstrate  that  baseline  nonrespondents  in  Project  ALERT  differed  from 
respondents  in  many  ways.  We  found  significant  differences  on  most  variables  we 
investigated,  often  surfacing  for  each  source  of  nonresponse.  We  observed  substantial 
differences  in  nonresponse  rates  among  schools,  among  racial/ethnic  groups,  and 
between  boys  and  girts.  Student  refusals  and  absentees  clearly  differed  from  respondents 
in  ways  the  literature  suggests.  These  groups  exhibited  much  lower  grades,  more 
absences,  and  higher  rates  of  cigarette  and  marijuana  use. 

2.  Nonresponse  had  little  Impact  on  the  sample.  Despite  the  often 
dramatic  differences  noted  above,  baseline  nonresponse  altered  sample  characteristics 
only  slightly.  For  example,  we  estimate  that  nonresponse  reduced  the  percentage  of 
nonwhite  students  by  only  0.9  percent — from  29.2  in  the  baseline  population  to  28.3 
among  respondents.  Similarly,  the  percentage  of  girts  dropped  by  only  1.1  percent  and 
mean  math  grade  point  average  rose  by  just  0.02  grade  points.  If  one  treats  student 
refusals  of  saliva  as  respondents,  the  gaps  become  even  smaller.  Clearly,  whether  the 
experiment  covered  a  population  of  interest  (external  validity)  depends  much  more 
critically  on  the  initial  selection  of  30  schools  than  on  baseline  nonresponse.  That  the 
experiment  retained  a  highly  diverse  sample  of  schools  and  students  means  the  results 
will  apply  in  a  wide  variety  of  environments. 

Two  facts  explain  the  small  impact  of  nonresponse  on  sample  characteristics. 

First,  baseline  nonrespondents  comprised  less  than  one-sixth  of  the  baseline  population. 

If  nonresponse  had  occurred  more  frequently,  the  potential  damage  would  have 
increased.  Second,  parent  refusals  touted  to  resemble  respondents  and,  consequently,  to 
dampen  or  even  counter  the  impact  of  student  refusals  and  absentees. 

We  expect  that  nonresponse  would  have  had  a  much  larger  impact  if  we  had  used 
active  parent  consent  The  literature  suggests  that  parent  refusals  from  an  active  consent 
procedure  would  liave  looked  something  like  our  student  refusals  and  absentees.  If 
parent  refusals  had  also  inflated  the  nonresponse  rate  near  50  percent  or  higher,  it  would 
have  seriously  damaged  the  experiment’s  external  validity.  That  parent  refusals  resulting 
from  active  consent  differ  from  those  resulting  from  passive  consent  is  not  surprising. 
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Refusals  to  return  active  consent  include  not  only  parents  who  do  not  want  their  children 
to  participate,  but,  apparently,  many  who  are  less  involved  in  their  children’s  education. 

Providing  makeup  surveys  to  spring  semester  absentees  also  helped  reduce  the 
size  of  differences  between  die  baseline  population  and  the  sample  of  baseline 
respondents.  Obviously,  this  step  reduced  the  number  of  nonrespondents.  Just  as 
important,  spring  semester  students  who  attended  makeup  sessions  at  baseline  reported 
wave  2  substance  use,  grades,  and  absences  similar  to  those  reported  by  students  who 
remained  absent. 

3.  Internal  validity  was  maintained.  The  need  to  assign  each  school  to  a 
single  treatment  condition  contributed  to  differences  among  treatments  in  nonresponse. 
Because  nonresponse  rates  differed  substantially  by  school,  cross-treatment  differences 
showed  up  despite  random  assignment  of  schools.  Similarly,  some  differences  remained 
across  treatments  in  the  characteristics  of  nonrespondents.  If  it  had  been  desirable  to 
assign  treatments  to  individual  students  (or  classes)  within  schools,  these  differences 
could  have  beat  reduced. 

Given  that  we  needed  to  assign  schools  to  treatments,  internal  validity  was  good. 
Little  evidence  exists  that  treatment  affected  either  the  rates  of  nonresponse  or  who 
responded.  Thus,  the  small  differences  among  treatments  appear  to  be  random  instead  of 
systematic.  Standard  statistical  methods  will  be  used  in  analysis  of  treatment  effects  to 
adjust  for  differences  caused  by  baseline  nonresponse  and  other  random  factors. 

4.  Steps  to  reduce  nonresponse  succeeded.  The  steps  taken  to  reduce 
baseline  nonresponse  in  Project  ALERT — passive  informed  consent,  careful  explanation 
of  data  privacy,  and  makeup  sessions — proved  successful.  They  held  the  combined  level 
of  baseline  nonresponse  to  16  percent  More  important  these  steps  sharply  limited 
differences  between  the  baseline  population  and  baseline  sample  and  avoided  substantial 
differences  among  the  treatment  groups. 
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